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Abstract 

In this study, the purpose was determine whether there was concordance among raters 
in the assessment of the writing performance using analytic rubric; furthermore, factors 
affecting the assessment process were examined. The analytic rubric used in the study 
consists of three sections and ten properties: External structure (format, spelling and 
punctuation), language and expression (vocabulary, sentences, paragraphs, and expressi- 
on), organization (title, introduction, story, and conclusion). The basis of the study is com- 
posed of narrative texts written by 200 students studying at the sixth and seventh grades 
of schools located on the Anatolian side of Istanbul (i.e., Beykoz, Kadikoy, Umraniye, and 
Uskudar). Texts were assessed in accordance with the analytic rubric by six raters. It was 
determined that the concordance among raters was sufficient according to the results of 
the assessment. 
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In primary education, writing is a skill area aiming to enable students to 
express their feelings, thoughts, imaginations, knowledge, ideas, and ob- 
servations on any topic in an efficient and planned manner. Considering 
its characteristics, it is difficult to assess this skill using multiple-choice 
questions. While the assessment of the written expression at sixth, se- 
venth, and eighth grades of the primary education was traditionally ba- 
sed on answering a minimum 30-point composition questions in written 
examinations, performance assessment has gained importance with the 
Primary Education Turkish Language Course (6 th , 7 th and 8 th Grades) 
Curriculum (MEB Talim ve Terbiye Kurulu Ba^kanhgi, 2007, p. 614). 

Performance assessment is an assessment that observes students’ pro- 
duction or participation within a process and is based on making 
judgments. It consists of two parts: task and scoring. One would not 
exist without the other and the performance cannot be assessed (Arter, 
Span del, Culham, & Pollard, 1994, p. 48). Writing task is determined 
and relevant framework is drawn by teachers as an element of perfor- 
mance assessment. It is determined what to score during the assessment 
process and a measurement tool is developed accordingly. The process or 
the product is assessed according to the structure of the task. 

When we take into consideration the number of students in classrooms 
under the conditions in Turkey, it may be thought that process assess- 
ment could be more difficult compared to the traditional assessment; 
however, it should not be disregarded that it would be more concrete 
and/or objective to assess the writing skill, or rather writing perfor- 
mance, compared to listening, speaking and reading skills, because it 
constitutes a proof for the assessment. 

The Curriculum aims to develop writing skills and achieve six main 
objectives and attainments. Consequently, in the assessment of the stu- 
dents’ writings, it requires preparation of measurement and assessment 
tools in line with the text linguistics, literature, and composition know- 
ledge within the framework of these objectives and attainments. 

The measurement and assessment tools aimed at students’ writings have 
similarities and differences depending on their objectives. Most of the 
measurement tools published with regard to written expression use a 
100-point scoring system. The measurement tool used in the Turkish 
Language (6 th , 7 th and 8 th Grades) Curriculum of the Primary Educati- 
on proposes assessment of student’s writings on the basis of a 100-point 
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scoring system. The basic problem with such scales is that it is not clear 
how to determine the top and bottom levels of a score (score distribu- 
tions) at a specific criterion; however, the explanation why a specific 
criterion is given a specific score is very important in terms of both the 
rater (teacher) and the rated (student) that creates the product assessed, 
as well as for the objectivity of the assessment. 

Rubrics 

Scoring guides called rubric are used in the assessment of written exp- 
ressions in many countries. “There are three types of rubrics used in 
the assessment of written expressions: Primary trait, holistic and analy- 
tic” (Weigle, 2002, p. 109). Primary trait rubrics are mainly used in the 
assessment of students’ basic writing skills relating to special writing 
tasks (for detailed information, see Weigle, 2002, pp. 109-112). Holistic 
rubrics are assessment tools used in the assessment of students’ works, 
aim to rate the properties with a score in line with the determined pro- 
perties, and define different levels of performance superficially (Elbow, 
2000; Gunning, 2006; Weigle, 2002). The holistic scoring performed 
in accordance with this assessment tool is controversial in terms of the 
fact that it involves giving a single score or point to the whole text, and 
combined measurement and assessment of different properties consti- 
tuting a writing. 

Because teachers/raters approach the text in a holistic manner in holis- 
tic scoring, they are affected by the whole text rather than being extre- 
mely affected by an important element in it (Gunning, 2006; Wolcott, 
8c Legg, 1998). However, when the writing is taken into consideration 
as a whole, some properties may become prominent and consequently 
raters might reward a not-so-good writing with a high score under the 
influence of these properties even if it is not worth it, or vice versa. 

Even though it is difficult to prepare a holistic rubric, assessments are 
easier and faster with this type of rubric; because, it requires assignment 
of a single score for the whole writing. “Holistic rubrics have limited 
capability to obtain complete and correct data from students. This is 
not a useful method particularly for development of students with low 
or medium level performance. The difficulty of preparation of a holistic 
rubric is distinguishing a level from another when all properties are lis- 
ted” (Martin- Kniep, 2000, p. 35). 
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Analytic rubrics are most suitable in distinguishing the analytic levels. 
The application of analytic scoring to writing means attempting to obta- 
in shared score sets to analyze the sub-skills in samples of each student’s 
writing (Babin, & Harrison, 1999, p. 117). Analytic scoring means sco- 
ring the properties/components (handwriting, sentences, title, etc.) 
constituting the writing. To this end, the properties, which the written 
expression should contain, are determined. The different property parts 
are evaluated separately. Consequently, assessments performed on the 
basis of the analytic rubric takes time; but “there is consensus among 
many scientists that analytic scoring is wider, more comprehensive, and 
useful” (Babin, & Harrison, 1999, p. 117). The comprehensiveness of 
the assessment tool is beneficial in terms of both teachers and students; 
because, teachers can better understand what score he/she will assess for 
specific properties according to specific criterion, and implements them 
accordingly. And if students know these, they can create a quality text 
according to criteria specified. 

According to Crehan (1997), analytic scoring provides guidance to te- 
achers for their teaching activities and to students for their writing ac- 
tivities, and provides them with more information about the current si- 
tuation, and owing to more orderly and comprehensive feedback, assists 
determination of the strong and weak aspects in students’ writings. 

Analytic or holistic rubrics can be used depending on the purpose of 
teachers and performance expected from students in the assessment of 
student’s writings. However, the matter as to which one is more reliable 
is controversial. Some researchers assert that analytic rubric is more re- 
liable than holistic rubric (e.g. Elbow, 2000; Gunning, 2006). 

“The reliability of scoring written expression activities is generally con- 
sidered lower than reliability of other written examinations scored with 
the same methods. The reason is that raters cannot reach an agreement 
on what properties should exist in a composition and that assessment is 
not performed with subjective criteria. To alleviate this drawback, it is 
considered to score compositions with analytic keys. Analytic key inc- 
reases the harmony among raters, therefore the reliability of scores in 
many cases” (Turgut, 1990, p. 65). The criteria that should be used in the 
assessment of students’ writings can be determined through compositi- 
on, text linguistics, and literature knowledge. 

The general objective of this study is to determine whether there are 
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objective criteria in the assessment of students’ writings. In other words, 
whether there is concordance among raters using the same measure- 
ment and assessment tool. In line with this general objective, answers 
were sought to the following questions: 

What is the rate of concordance among raters using the same scale? 

What are the factors that lead to probable differences in assessment? 

What is the level of concordance among raters in terms of the external 
structure, language, expression, and organizational parts of the texts be- 
ing assessed? 


Method 

The survey method was used in this study that aimed to determine the 
reliability of the analytic rubric. Relational survey methods aim to de- 
termine the existence and/or the level of any changes between/among 
two or more variables. Relational survey is performed by way of corre- 
lation or comparison (Karasar, 2002, p. 81). The category analysis from 
the document analysis and content analysis is used for the analysis of 
the data. 

Research Population 

The study covers a total of 200 texts randomly selected from writings 
created by 623 students studying at the sixth and seventh grades of five 
schools in Istanbul in the 2007-2008 school year. 

To increase reliability and prevent any probable subjectivity, it was 
decided to include more than two raters in the study. Six raters who 
participated in the study voluntarily constitute the research population. 
The functions, titles, genders, and periods of service of the raters are as 
follows: 

1 st Rater: master, (male, 8 years) 

2 nd Rater: master (male, 8 years) 

3 rd Rater: master (female, 7 years) 

4 th Rater: master (female, 5 years) 

5 th Rater: Turkish Language Teacher (female, 6 years) 

6 th Rater: Turkish Language Teacher (female, 2 years) 
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Data Collection Tools 

The data collection tools used in the study are memoir type of writings 
created by students with narrative expression as well as the analytic 
rubric used in their assessment. Because it would be relatively easy to 
write, students were requested to write an event, which they witnessed 
or were affected by. Properties constituting the analytic rubric were de- 
veloped through the examination of the criteria, and the objectives and 
attainments in the curriculum, literature surveys, and receiving expert 
opinions. 

Process 

Texts were assessed by 6 expert raters in accordance with the analytic 
rubric. 

Training Provided to Raters: Raters were provided two sessions of trai- 
ning to ensure performance of assessments in accordance with the rub- 
ric. 

Assessment stages can be summarized as follows: 

(i) Before reading a text written by a student, the spaces within and 
out of the text area are paid attention in terms of the format pro- 
perties in order not to be influenced by things expressed in the text 
and by any other properties of the text. 

(ii) The entire text is read quickly. During this reading, rater tries to get 
accustomed to the writing of the student. He/she tries to unders- 
tand the subject and event explained in the text. By the end of the 
first reading, any illegible or difficult-to-read words - if any - are 
identified. Format properties are scored. Meanwhile, rater forms 
an opinion about the narration of the student. The selection of 
vocabulary, length of sentences, paragraph integrity, punctuation, 
and spelling errors are taken into consideration, but they are not 
scored. 

(Hi) During the second reading, the introduction, story, and conclusion 
sections are read and scored. Finally, the title is given a score con- 
sidering how much it reflects the subject of the writing as a result 
of which the assessment of the organization properties completes. 

(iv) During the third reading, the vocabulary, sentence, and paragraph 
properties are scored. At this stage, attention is paid to determiners 
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and the use of cliches as well as sentence structures, order of sen- 
tences, and paragraph integrity. 

(v) The text is skimmed to score the spelling and punctuation proper- 
ties, and taking into consideration the impressions obtained during 
previous readings, observance of spelling and punctuation rules is 
scored using the error counting method. 

(vi) Finally, the assessment process is completed by scoring the narra- 
tion (style of narration, type information, and style) properties. 

Opinions were exchanged with the raters whenever required. No time 
limitation was imposed to prevent random mistakes and the raters from 
getting bored during the assessment process. Participants read and as- 
sessed the texts in the order they were given. 

The category analysis method was used for the analysis of the data. The 
analysis categories may be based on a wide variety of dimensions. Di- 
mensions relating to the thing conveyed by the message such as the sub- 
ject, direction of the message, the values it carries, goals and intentions, 
ways applied to attain the goals, properties used in description of per- 
sons, source of the message, person or audiences addressed by the mes- 
sage, place and time of the events, conflict subjects, etc., as well as the 
dimensions relating to the style of expression of the message such as the 
type, format of the message, grammatical properties of communication, 
syntax, rhetorical methods, etc. may be taken as basis in categorization 
(Bilgin, 1995, p. 96-97). 

To determine the scoring relating among the raters, the Pearson corre- 
lation analysis and reliability analysis were conducted using the SPSS 
11.5 software. The alpha value was also used, due to the fact it had been 
used in a previous similar study (Tezci, 2002). “The alpha coefficient 
does not give the reliability coefficient, but just indicates that reliability 
would be far higher than this” (Aker, Diindar, & Pe§ken, 2005, p. 55). 

Kendall’s W, also known as Kendall’s coefficient of concordance is also 
used to determine the concordance among the raters. “Kendall’s coef- 
ficient of concordance is used for measurement of concordance among 
raters” (Legendre, 2005; Powers, & Harris, 1991; Sigler, & Tallent- 
Runnels, 2006). According to Aker et al. (2005, p. 53), this coefficient is 
used in cases where there are more than two raters. “The purpose of this 
measurement is not to determine whether raters make the assessment 
correctly, but to determine if there is concordance among them” (Po- 
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wers, 8c Harris, 1991). When this coefficient is between 0 and 1, it me- 
ans there is concordance, when it is 0, it means there is no concordance 
among raters (Legendre, 2005; Malvern, 8c Skidmore, 2001). 

Kendall’s coefficient of concordance (w) is calculated as follows (Le- 
gendre, 2005, p. 229): 


W = 


12 S 

p 2 (n } -n)-pT 


5 = -Rf 


Results 

In the assessment made to the analytic rubric, the Pearson correlati- 
on analysis was conducted to determine concordance among six raters, 
and Friedman and Kendall coefficients of concordance and alpha values 
were calculated. 


Table 7 . 

Result of the Inter-rater Reliability Analysis 






Properties 

Highest 

Correlation 

(Pearson) 

Raters 

Lowest 

Correlation 

(Pearson) 

Raters 

Average 

Correlation 

(Pearson) 

Freidman 
coefficient of 
concordance (w) 

Kendall 
coefficient of 
concordance (w) 

Alpha 

Value 

1.1. Format 

.9361 

1-6 

.6855 

2-4 

.8104 

.0365 

.001922 

.9626 

1.2. Spelling & 
punctuation 

.8950 

1-6 

.6385 

2-3 

.7515 

.0049 

.001251 

.9480 

2.1. Vocabulary 

.8612 

1-5 

.5307 

2-4 

.7064 

.0131 

.001084 

.9332 

2.2. Sentences* 

.8338 

1-5 

.5251 

2-3 

.6728 

.0271 

.001015 

.9222 

2.3. Paragraphs 

.9350 

1-6 

.7470 

2-3 

.8300 

.0219 

.002020 

.9663 

2.4. Narration * 

.7883 

2-5 

.5335 

2-4 

.6599 

.0318 

.001125 

.9196 

3.1. Title 

.8931 

1-6 

.5717 

3-6 

.7166 

.0095 

.001188 

.9365 

3.2. Introduction 

.9068 

3-4 

.6061 

2-3 

.7054 

.0019 

.001185 

.9348 

3.3. Story* 

.8269 

1-6 

.5627 

2-3 

.6841 

.0181 

.001295 

.9279 

3.4. Conclusion 

.8498 

1-6 

.5513 

2-3 

.7128 

.0091 

.001345 

.9364 


When we examine the table, according to the result of the inter-rater 
reliability analysis, it can be said that there is very favorable concordance 
with respect to the properties of format (.8104) and paragraph (.8300) 
according to the average correlation among six raters. According to the 
average correlation among six raters there is a relation at high correla- 
tion among the properties of spelling and punctuation (.7515), vocabu- 
lary (.7064), title (.7166), introduction (.7054) and conclusion (.7128). 
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As for the sentence (.6728), narration (.6599) and story (.6841) proper- 
ties, according to the average correlation among six raters, existence of a 
correlation average below 70 indicates a medium level concordance. 

The alpha value dropped or increased in direct proportion to the Pear- 
son correlation results. Only if the alpha value is not close to 1 in any of 
the properties can give a positive impression with regard to reliability. 
In scores given to properties constituting the rubric, Friedman’s reliabi- 
lity coefficient results between 0 and .05 indicates concordance among 
raters. 

And Kendall’s coefficient of concordance results between 0 and 1 also 
indicates concordance among raters. 

The medium level of concordance among six raters with respect to the 
sentence, narration, and story properties in the analytic rubric accor- 
ding to the findings of the Pearson correlation average is below the 
statistically expected relation (.70). Some researchers assert that a cor- 
relation between .3 and .7 is a medium-level relations (e.g. Saruhan, & 
Ozdemirci, 2005, p. 39), and some other assert a relation between .65 
and .85 would be sufficient in social sciences studies (e.g. Cohen, & 
Manion, 1994, p. 139-140). Consequently, a correlation above .65 for 
three properties (sentence, narration and story) among ten properties 
in the rubric, and above .70 for all other properties is sufficient in terms 
of reliability for social sciences - particularly in qualitative assessments 
particularly such as written expression. The level descriptions in the rub- 
ric, as well as participation of expert raters in the assessment process, 
were effective in achievement of this sufficiency. 

Reasons of reaching these findings based on the inter-rater concordan- 
ce analyses, observations, and theoretical knowledge can be listed as 
follows: 

1.1. It can be possible to make an assessment about the format pro- 
perties at a glance. Therefore, a high concordance among raters is an 
expected result. 

1.2. The assessment carried out within the scope of this study was per- 
formed by counting spelling and punctuation errors. It can be said that 
the performance of this assessment is not difficult as a result of a careful 
examination. 

2.1. Words are distributed throughout the text, for words are the cons- 
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tituent of texts. When the whole text is read and a value from the four 
levels is required to be given to the vocabulary properties, it will be 
necessary to look at the qualifiers, and at the status of nouns and verbs, 
and the assessment should be carried out accordingly. 

2.2. It may be possible to define the sentence properties with a single 
sentence during the preparation of the scale; but this definition would 
turn into a totally holistic nature. Now that sentence properties can be 
analyzed, it is seen that it is necessary to conduct a theoretically analytic 
assessment. It is possible to make modifications on sentence properties 
included in the analytic rubric. 

2.3. Even though the paragraph properties can be distributed throug- 
hout the text, they are less in number compared to grammatical rules or 
punctuation marks, words, and sentences. Therefore, carrying out an as- 
sessment based on these properties may be easier. However, the assess- 
ment of subject unity in individual paragraphs or consistency between 
paragraphs may be difficult because, when the texts within the scope of 
this study are examined, it is seen that some students based their wri- 
tings on three paragraphs taking into consideration the introduction, 
story, and conclusion section. Even though introduction and story pa- 
ragraphs were consistent in themselves, there were texts in which it was 
possible to divide story section into several paragraphs. High number 
of texts consisting of single paragraph facilitated assessment of parag- 
raphs. 

2.4. When the narration properties in the analytic rubric are examined, 
it can be seen that there are many disjunctive elements. Furthermore, 
the possibility that the level of concordance could be low during prepa- 
ration of the rubric because this property reflects the writer’s (student’s) 
style in the text, and because the style carries individuality and subjecti- 
vity, and subjectivity in terms of raters, hence it changes from one person 
to another, was a result anticipated by the researchers. Therefore, it was 
seen that concrete properties should be assessed in texts in an attempt 
to minimize incongruence, and to this end, extent of use of emotions, 
thoughts and interpretations by writers, extent of conformance to type 
of text, and whether events and situations are explained in an organized 
manner are assessed in the analytic rubric. 

3.1. Fixed location of titles and the fact that they consist of a low number 
of words may facilitate assessments. However, effectiveness of titles may 
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change from one person to another. Although it had been anticipated 
during preparation of the rubric that this property would be concordant, 
even at the highest level, among raters, it did not prove true. 

3.2. Even though the properties of the introduction section are concrete, 
the sentence elements specified in property definitions (time, place, per- 
son, event information) maybe placed in the story section depending on 
the viewpoint of the writer (student), or intentionally or unintentionally 
on the preference of the writer. For this reason, they may be unnoticed 
by the rater or raters might display different attitudes in assessing these 
properties. Furthermore, the effectiveness, accuracy/exactness or defini- 
teness of presentation of these elements may be debatable. 

3.3. As it is known, in terms of organization, the story sections are the 
longest sections of texts. Dialogues and details in the text influence the 
rater. In this respect, it was seen that a general evaluation could be made 
in assessment of the conclusion section. 

3.4. The conclusion section is related to whether things expressed in the 
text have concluded or not, and whether the writer has added his/her 
thoughts or comments. Therefore, it can be said that assessment of the 
conclusion section is not difficult. 

In order to be able to better understand the assessment process, the 
correlation results of the first 50 and last 50 texts scored by raters were 
compared. 


Table 2. 

Inter-rater Concordance Averages 


Correlation averages 

Correlation averages 

Properties 

obtained as a result of 

obtained as a result of 

assessment of the first 50 

assessment of the last 50 


texts (Pearson) 

texts (Pearson) 

1.1. Format 

.6244 

.8937 

1.2. Spelling & punctuation 

.7029 

.8308 

2.1. Vocabulary 

.6391 

.7686 

2.2. Sentences* 

.5098 

.7601 

2. 3. Paragraphs 

.6867 

.8784 

2.4. Narration * 

.4288 

.7543 

3.1. Title 

.6663 

.7895 

3.2.1 ntroduction 

.5394 

.8035 

3.3.Story‘ 

.5681 

.8024 

3.4. Conclusion 

.5330 

.7986 
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As it can be seen in Table 2, when the inter-rater concordance averages 
are examined, concordance among raters is low in the assessment of the 
first 50 texts, while the rate of concordance increases in the assessment 
of the last 50 texts. Based on these findings, it is possible to say that 
raters are more concordant in the last texts compared to the first texts in 
the process of scoring according to the properties existing in the rubric. 
Participants may be getting accustomed to the rubric and assessment 
may be shown as a reason of this finding. 

When the Table 1 is examined, it is seen that there are less concordance 
between the first and third rater compared to others. In the interview 
with the second rater, he stated that he took long breaks between text 
assessments. Emphasizing that he was sometimes influenced by the to- 
pic and therefore gave higher ratings, the rater said sometimes he scored 
under positive or negative influence of a specific section of texts. 

It is seen that the third rater gave lower ratings particularly to exter- 
nal structure, language, and expression properties and higher ratings to 
organization properties. In the interview conducted, she was asked the 
reason of this. She stated that she did not find the external structure and 
language properties sufficient but that she found the topics interesting. 

The third and fourth raters were affected by the topic of writings writ- 
ten mainly by female students (birthday celebrations, year-end picnics 
and parties, death, etc.) and therefore gave higher ratings particularly 
to organization properties compared to the other raters. Both raters are 
female and it is understood that they were affected by emotional texts 
written by female students. 

Fourth rater stated that she noticed that when she looked at the ratings 
she gave during the scoring process, she scored the vocabulary and sen- 
tence properties in direct proportion to each other. It is found that the 
fifth rater also used a similar scoring method. 

The fifth and sixth raters who are Turkish language teachers gave lo- 
wer ratings particularly to the spelling and punctuation properties. It 
is observed that the fifth rater displayed a medium-level concordance 
relation with the other raters in most of her scorings. 

Because the first and the sixth raters acted jointly in preparation of the 
rubric and collection of data, and performed assessments jointly before 
the 200 texts, they were more accustomed to the rubric compared to the 
others. Therefore, the higher level of concordance between these raters 
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in scoring of many properties can be understandable. 

It can be said that the lengths of texts are important in assessments 
carried out by raters because raters generally gave lower ratings to short 
texts. However, this does not mean that all long texts were scored hig- 
her. 


Discussion 

When the text structures and the entire composition concept are taken 
into consideration and analytic rubric is examined, it can be seen that 
the spelling and punctuation, vocabulary, sentences, paragraphs, expres- 
sion, and story properties are more comprehensive or multi -dimensional 
compared to format, title, introduction, and conclusion properties and 
that analysis and assessment of these properties should be made very 
carefully. 

In assessments performed in accordance with the analytic rubric, a suf- 
ficient level of concordance has been observed among raters in terms of 
social sciences studies. 

During the assessment process, raters scored properties different from 
the rubric for a variety of reasons (e.g. time, failure to concentrate, exp- 
ression of writer, attractiveness of topic, differences between texts). High 
number of texts and the fact that source of texts is two different grades 
may have played an important role in this situation. 

Some descriptions relating to the criteria existing in the analytic rubric 
display holistic characteristics. This is related to both written expression 
-in the first place- and the rubric because “some components consti- 
tuting the analytic rubric may have holistic characteristics” (Babin, 8c 
Harrison, 1999, p. 118). 

When the number of raters participating in the assessment, the number 
of texts assessed, and the structure of written expression and qualitative 
analysis are taken into consideration, it is seen that the ten properties 
and the concordance among raters in the assessment carried out in line 
with the analytic rubric are sufficient. 

The analytic rubric should be used for the assessment of written expres- 
sions. Thus, it allows teachers to determine the deficiencies in students’ 
writing skills right at the beginning of the school year, to act in line with 
these deficiencies, and to adopt an appropriate strategy. 
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Rubrics developed in accordance with the analytic rubric preparation 
principles should be applied at schools. The Turkish language teachers 
and classroom teachers should test the practicability of the rubrics thro- 
ugh scoring trials. 

Teacher guidance books rather contain holistic rubrics for assessment of 
written expressions. However, analytic rubrics must also be included. 

The organization properties (introduction, story, conclusion sections) in 
the analytic rubric prepared for the purpose to assess the narrative texts 
used in this study may be modified for use in assessment of other types 
of texts (e.g. informative texts). 
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